Managing Resource Groups#

Overview#

This guide provides step-by-step instructions for managing resource groups in DPS.

For more information, refer to Resource Groups.

Note: Resource groups are typically managed by a workload scheduler. This guide demonstrates how to manage them directly.

Prerequisites#

  • DPS server running and accessible

  • dpsctl installed and authenticated

  • Active topology with entities configured

Basic Resource Group Workflow#

Step 1: Create Resource Group#

Shared GPU mode defaults to on. The following command creates a shared-GPU resource group. Pass --shared-gpu-enable=false if you need per-node GPU caps instead of a group-wide GPU budget.

dpsctl resource-group create \
  --resource-group "ml-training-job" \
  --external-id 12345 \
  --policy-bundle MaxP

--priority is optional and defaults to 0, the highest precedence. A larger number is lower importance during stealing. Refer to Resource Group Priority.

Step 2: Add Hardware Resources#

dpsctl resource-group add \
  --resource-group "ml-training-job" \
  --entities "node001,node002,node003"

Step 3: Activate Power Policies#

dpsctl resource-group activate \
  --resource-group "ml-training-job"

Step 4: (Optional) Configure Per-GPU Power Limits#

After activation, use the standalone GPU policy command for fine-grained GPU control on a node in the active resource group:

dpsctl gpu-policy \
  --node "node001=500,550,600,700,650,700,550,600"

Step 5: Cleanup#

dpsctl resource-group delete \
  --resource-group "ml-training-job"

Dynamic Resource Management#

Resources can be added to or removed from resource groups at any time, including after activation. This enables dynamic workload scaling without full resource group deactivation.

Note: Add/remove operations are rejected while a resource group is activating or deactivating. Wait for the operation to complete before modifying resources.

Adding Resources to an Active Resource Group#

When adding resources to an active resource group, policies are applied immediately and power is reallocated as needed:

dpsctl resource-group add \
  --resource-group "ml-training-job" \
  --entities "node004,node005"

Strict Policy Enforcement#

Use --strict-policy to ensure the configured policy bundle is applied exactly. If power constraints prevent the bundle from being satisfied, the operation fails rather than retrying at the effective policy floor:

dpsctl resource-group add \
  --resource-group "ml-training-job" \
  --entities "node004" \
  --strict-policy

Controlling Power Reprovisioning#

By default, power may be redistributed from other resource groups if needed (power stealing). Use --allow-reprovision=false to prevent this:

dpsctl resource-group add \
  --resource-group "ml-training-job" \
  --entities "node004" \
  --allow-reprovision=false

Removing Resources from an Active Resource Group#

When removing resources from an active resource group, remaining topology limits are restored. If no effective limit remains, the idle-policy or hardware-minimum release path applies as described in Policy Lifecycle and Fallback:

dpsctl resource-group remove \
  --resource-group "ml-training-job" \
  --entities "node004,node005"

Troubleshooting#

Shared GPU Budget Rejections#

GPU policy updates on a shared-GPU resource group can fail even when each node’s requested watts look valid in isolation. DPS first fits the post-update group total to the resource-group GPU budget, then checks non-compute topology ancestors such as racks and PDN devices. Refer to Shared GPU Budget for the model.

Typical diagnostics include:

  • shared-GPU update violates topology constraint at device:<ancestor>

  • response violates topology constraints

The named ancestor is the topology entity whose load would exceed its constraint. Correct the request by lowering the requested GPU caps, spreading the increase across more members, or raising the applicable topology limit, then retry the update.

Verifying Node Status#

After activating a resource group, always verify that each individual node’s policy was applied successfully:

  1. Check that each node in node_statuses has "ok": true

  2. Review any error messages in diag_msg fields

  3. If any nodes show "ok": false, investigate the specific error before proceeding

Example of a failed node:

"node004": {
  "status": {
    "ok": false,
    "diag_msg": "BMC connection timeout"
  }
}

PRS#

Power Reservation Steering (PRS) is a product optionally included and directly integrated with DPS that performs real-time power allocation adjustment for resource groups. PRS consumes telemetry provided by DPS to generate recommendations for power allocations at a per-GPU level, which DPS validates against the PDN constraints before applying. PRS can be controlled in a few ways via DPS:

  • Not deployed (configured via Helm chart values option .Values.prs.enabled)

  • Toggled via global configuration in the DPS WebUI

  • Toggled per ResourceGroup during creation with option --prs-enabled=false It is recommended to deploy PRS and leave it enabled globally. For Max-P Resource Group, we recommend disabling PRS and leaving it on for all others.

Further Reading#