Power Reservation Steering (PRS)

Overview

Power Reservation Steering (PRS) is an advanced dynamic power management feature in DPS that automatically redistributes power across compute nodes based on real-time telemetry data. PRS optimizes power allocation within a power domain by monitoring GPU utilization and dynamically adjusting power limits to maximize workload performance while staying within overall power budgets.

Unlike static power policies that set fixed power limits, PRS continuously analyzes workload behavior and steers unused power from idle or underutilized GPUs to those that need more power, enabling more efficient use of datacenter power capacity.

Key Concepts

Dynamic Power Redistribution

PRS operates in a continuous loop that:

1. Collects Telemetry - Gathers real-time power usage and GPU utilization data from all nodes in the domain
2. Analyzes Workload - Identifies GPUs that are underutilizing their allocated power
3. Calculates Recommendations - Determines optimal power distribution based on actual demand
4. Applies Changes - Dynamically adjusts GPU power limits through DPS

Budget and Idle-Power Semantics

PRS works within an aggregate power budget. The budget is an upper bound, not an exact draw target, and PRS does not try to make every node consume the same amount of power. Nodes can receive different GPU limits as their observed demand changes.

Idle and underutilized GPUs still retain the minimum power defined by their device specification. DPS 0.8.x does not expose a separate workload-idle threshold that can be tuned per node, resource group, or topology. This is different from a device specification’s idlePolicy: idlePolicy is a fallback policy used when no explicit topology policy is assigned, not the telemetry threshold PRS uses to classify current workload demand.

Integration with Resource Groups

PRS integrates seamlessly with DPS’s resource group system. When a resource group is activated with PRS enabled:

• The resource group’s nodes are added to the PRS domain
• PRS begins monitoring and optimizing power for those nodes
• When the resource group is deactivated, nodes return to their topology defaults

Configuration Methods

PRS can be configured at multiple levels, providing flexibility for different deployment scenarios.

1. Helm Chart Configuration

Enable PRS deployment and connection when installing DPS via Helm:

# Enable PRS deployment
prs:
  enabled: true
  image:
    repository: nvcr.io/nvidia/prs
    tag: 1.0.12-prs-0.1.1.10-dpsapi
  config:
    configServerPort: 8880
    jobSchedServerPort: 8881
    # Chart default; tune from measured BMC latency.
    loopIntervalSeconds: 10
    rpcTimeoutSeconds: 10
    windowSize: 3

# Connect DPS-Server to PRS
dps:
  prs:
    enabled: true
    headroomPercent: -30.0
    # hostPort defaults to internal PRS service if not specified
    hostPort: ""

Both enable switches are required. prs.enabled deploys the PRS service, while dps.prs.enabled connects DPS-Server to that service and supplies the startup headroom value.

loopIntervalSeconds: 10 is the chart default, not a fixed recommendation for every deployment. Set the interval from measured BMC latency so PRS does not iterate faster than the BMC telemetry and power-control paths can respond. Follow the MaxLPS pilot BMC latency measurement procedure and its BMC latency tuning instructions before choosing the production value.

PRS Configuration Parameters

How Headroom Is Calculated

Headroom is global to the DPS deployment. It is not independently configurable per node, resource group, or topology. For each node in an active PRS-enabled resource group, DPS calculates:

gpuPool = max(0, nodePolicyCap - reservedCPU - reservedMemory)
nodeBudget = gpuPool * (1 + headroomPercent / 100) + nonGPUStatic
domainBudget = sum(nodeBudget for every node in the resource group)

nodePolicyCap is the node-level limit from the effective policy, with the node hardware maximum as the fallback when no node limit is present. reservedCPU and reservedMemory come from the policy when present and use the hardware-minimum fallback otherwise. DPS calculates nonGPUStatic = max(0, nodeHardwareMinimum - gpuCount * gpuMinimum), so the GPU minimum is not counted twice. Adding nonGPUStatic after the multiplier means headroom scales only the GPU pool, not the chassis baseline.

The valid range is -50 through 100:

• A negative value reduces the GPU pool sent to PRS and causes more aggressive capping. Negative headroom is generally recommended.
• 0 sends the unscaled GPU pool.
• A positive value gives PRS more redistribution slack.

Negative headroom is important to PRS responsiveness because it keeps the domain under an active aggregate constraint. When workload demand moves between GPUs, PRS must actively steer the constrained budget instead of absorbing the change in unallocated slack. The -30 setting has been tested and is the recommended starting point for DPS 0.8.x. Zero and positive values are valid, but the additional slack can make steering less responsive; use them only after validating power compliance and workload performance. Headroom controls the strength of this budget pressure, while loopIntervalSeconds separately controls how often PRS reacts.

For example, if nodePolicyCap is 12,000 W, the CPU and memory reservations are 1,000 W and 500 W, and nonGPUStatic is 400 W, then gpuPool is 10,500 W. At -30, that node contributes 10,500 * 0.70 + 400 = 7,750 W to the domain budget. Four identical nodes contribute 31,000 W.

Headroom changes only the aggregate domain power_budget; it does not scale each GPU’s power_max, which remains the hardware TGP sent to PRS. It is also separate from topology PowerDomain.OverProvisioningPercentage. PRS headroom changes the runtime GPU budget sent to PRS, while topology overprovisioning changes how much capacity DPS can provision above a PowerDomain operating limit.

The domain budget is an aggregate upper bound; it is not a target that every node must draw and it does not guarantee equal node power. DPS still validates PRS recommendations against the effective resource-group policy, shared-GPU budget, PowerDomain, and NvGrid constraints. Headroom does not override those limits, and an infeasible budget or recommendation can be rejected.

Startup Defaults and the Helm Zero Limitation

The DPS 0.8.x Helm chart defaults dps.prs.headroomPercent to -30. The DPS-Server binary itself defaults --prs-headroom-percent (and PRS_HEADROOM_PERCENT) to 0 when it is run directly without the chart.

In the 0.8.x chart, setting the Helm value to the numeric value 0 still renders PRS_HEADROOM_PERCENT=-30.0 because the template treats numeric zero as empty. To use zero headroom, deploy DPS and then set the runtime value:

dpsctl settings update --set prs_headroom_percent=0

The runtime value is persisted. It takes precedence over Helm or direct-binary startup values on subsequent DPS-Server restarts. To return to another value, update the runtime setting explicitly; changing only the Helm value does not replace a value already persisted in the database.

Headroom Tuning

Use negative headroom for normal PRS operation. For an initial MaxLPS pilot, start with the tested chart default of -30. If the workload has unacceptable performance, repeat the same workload at -15. Keep the resource-group policy, topology, workload, load-generator mode, controlled input, and measurement window unchanged so the result can be attributed to headroom. Use the same target-QPS schedule for an open-loop test or the same configured concurrency for a closed-loop test. The topology’s PowerDomain.OperatingLimit defines the pilot constraint, the resource-group policy sets the starting requested allocation inside that constraint, and PRS headroom scales the GPU budget sent to PRS. Change only one control at a time.

2. Global Config API

Read or update the global PRS enable and headroom settings using the Global Configuration API:

Get Current Configuration

# gRPC request
grpcurl -plaintext localhost:50051 nvidia.dcpower.v1.ConfigurationManagementService/GetConfiguration

Response:

{
  "status": { "ok": true, "diagMsg": "Success" },
  "settings": {
    "prs_enabled": "true",
    "prs_headroom_percent": "-30",
    "metrics_interval_seconds": "30"
  }
}

Update PRS Setting

# Enable PRS
grpcurl -plaintext -d '{"settings": {"prs_enabled": "true"}}' \
  localhost:50051 nvidia.dcpower.v1.ConfigurationManagementService/UpsertConfiguration

# Disable PRS
grpcurl -plaintext -d '{"settings": {"prs_enabled": "false"}}' \
  localhost:50051 nvidia.dcpower.v1.ConfigurationManagementService/UpsertConfiguration

# Change global PRS headroom
grpcurl -plaintext -d '{"settings": {"prs_headroom_percent": "-15"}}' \
  localhost:50051 nvidia.dcpower.v1.ConfigurationManagementService/UpsertConfiguration

3. dpsctl Command Line

Manage PRS settings using the dpsctl command-line tool:

List Current Settings

dpsctl settings list

Output:

{
  "settings": {
    "metrics_interval_seconds": "30",
    "prs_enabled": "false",
    "prs_headroom_percent": "-30"
  },
  "status": {
    "diag_msg": "Success",
    "ok": true
  }
}

Enable PRS Globally

dpsctl settings update --set prs_enabled=true

Disable PRS Globally

dpsctl settings update --set prs_enabled=false

Change PRS Headroom

dpsctl settings update --set prs_headroom_percent=-15

Use dpsctl settings list to read the persisted value that DPS is currently using. Do not infer the effective value from the Helm values file alone.

4. User Interface (UI)

The DPS web interface provides a visual way to manage PRS:

1. Navigate to Settings in the DPS UI
2. Locate the Dynamic Power Integrations section
3. Find the Power Reservation Steering (PRS) card
4. Toggle the switch to enable or disable PRS
5. Enter a value from -50 through 100 in PRS Headroom (%)
6. Click Save to apply both settings

The UI displays PRS with descriptive tags:

• workload-agnostic - Works with any workload type
• topology - Applies across the entire topology
• resource-group - Integrates with resource group management

The headroom field is disabled while the global PRS switch is off. Values saved from the UI use the same persistent runtime setting as the API and dpsctl.

Applying, Verifying, and Rolling Back a Runtime Change

A runtime headroom update does not require a DPS-Server restart. When the effective value changes, DPS persists the value, updates its PRS client, and asynchronously recomputes and re-upserts every active PRS-enabled resource-group domain. The API or dpsctl command can therefore return before all domain upserts finish. A write of the existing value is a no-op and does not produce a new re-upsert or configuration-change log message.

After changing the value:

1. Confirm the persisted effective setting:

   dpsctl settings list

2. Confirm that the DPS-Server observed the change and initiated the re-upsert:

   kubectl logs -n dps -l app=dps-server -c dps-server --since=5m \
     | grep "PRS config changed, re-triggering PRS upsert"

3. Confirm the intended resource groups are active and have PRS enabled:

   dpsctl resource-group list

Record the previous value before tuning. Roll back by writing that value through the same runtime path, for example:

dpsctl settings update --set prs_headroom_percent=-30

Then repeat the verification steps. Rollback is also asynchronous and remains persisted across restarts.

5. Resource Group Configuration

Control PRS behavior on a per-resource-group basis:

Create Resource Group with PRS Disabled

dpsctl resource-group create \
  --resource-group "static-training" \
  --external-id 12345 \
  --policy "Node-High" \
  --dpm-enable=false \
  --prs-enabled=false

Create Resource Group with PRS Enabled (Default)

# PRS is enabled by default when creating resource groups
dpsctl resource-group create \
  --resource-group "dynamic-inference" \
  --external-id 12346 \
  --policy "Node-Med" \
  --dpm-enable \
  --shared-gpu-enable

PRS vs Dynamic Power Management (DPM)

DPS supports two complementary dynamic power features:

Both can be used together:

• DPM enabled allows DPS to dynamically adjust policies
• PRS enabled allows automatic power redistribution based on telemetry

# Full dynamic power management
dpsctl resource-group create \
  --resource-group "full-dynamic" \
  --external-id 12347 \
  --policy "Node-High" \
  --dpm-enable=true \
  --prs-enabled=true

# Static policy only (no dynamic adjustment)
dpsctl resource-group create \
  --resource-group "strict-policy" \
  --external-id 12348 \
  --policy "Node-Med" \
  --dpm-enable=false \
  --prs-enabled=false

Architecture

PRS Service Components

PRS runs as a separate service in the dps namespace with the following components:

1. Configuration Server (port 8880) - Manages PRS configuration and domain definitions
2. Controller Loop - Continuously monitors and optimizes power distribution

Data Flow

┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐
│   DPS-Server    │────▶│   PRS Service   │────▶│   DPS-Server    │
│  (Telemetry)    │     │  (Optimization) │     │ (Apply Limits)  │
└─────────────────┘     └─────────────────┘     └─────────────────┘
        │                       │                       │
        ▼                       ▼                       ▼
   GPU Metrics            Power Budget           GPU Power Limits
   Collection             Calculation            Application

Best Practices

When to Enable PRS

• Heterogeneous workloads - When GPUs in a domain run different workload types
• Variable utilization - When GPU utilization fluctuates over time
• Power-constrained environments - When maximizing work within a fixed power budget is critical
• Large-scale deployments - When manual power tuning is impractical

When to Disable PRS

• Consistent workloads - When all GPUs run identical, steady-state workloads
• Fixed-cap requirements - When a workload must keep a fixed policy cap without dynamic redistribution
• Debugging scenarios - When isolating power-related issues
• Short jobs - When job duration is shorter than PRS optimization cycles

Configuration Recommendations

1. Start with defaults - PRS default settings work well for most deployments
2. Monitor before tuning - Observe PRS behavior before adjusting parameters
3. Use per-resource-group control - Disable PRS for specific workloads that need strict power limits
4. Coordinate with policies - Ensure power policies provide reasonable bounds for PRS optimization

Troubleshooting

Headroom Value Is Rejected

prs_headroom_percent must be a finite number from -50 through 100. Values outside that range are rejected by the runtime API and dpsctl.

Effective Headroom Does Not Match Helm

Run dpsctl settings list. A previously saved API, dpsctl, or UI value is persisted and takes precedence over the Helm startup value. Update the runtime setting explicitly to change it. If Helm numeric zero rendered as -30 on DPS 0.8.x, use the documented runtime prs_headroom_percent=0 workaround.

Headroom Changed but PRS Behavior Did Not

Confirm that the PRS service is deployed, the effective global prs_enabled setting is true, and the active resource group has PRS enabled. When the effective headroom value changes, the domain re-upsert is asynchronous, so also inspect the DPS-Server log for completion or failure after the configuration-change message. A no-op write of the current value does not produce a new message or re-upsert. A strongly negative value can make the computed domain budget lower than PRS’s node and GPU minimums; correct the policy or increase headroom if PRS rejects that configuration. A positive value still cannot bypass policy, shared-GPU, PowerDomain, or NvGrid enforcement.

PRS Not Redistributing Power

1. Check PRS is enabled globally:

   dpsctl settings list

2. Verify PRS service is running:

   kubectl get pods -n dps -l app=prs

3. Check resource group PRS setting:

   dpsctl resource-group list

PRS Service Connection Issues

1. Verify DPS-Server can reach PRS:

   kubectl logs -n dps statefulset/dps-server | grep -i prs

2. Check PRS configuration:

   kubectl get configmap -n dps dps-prs-config -o yaml

Further Reading

• Power Policies - Define static power limits
• Resource Groups - Manage workload power allocations
• Workload Power Profiles Settings - GPU-specific power optimization
• Topologies - Datacenter infrastructure modeling