Managing Resource Groups

Overview

This guide provides step-by-step instructions for managing resource groups in DPS.

For more information, see Resource Groups.

Note: Resource groups are typically managed by a workload scheduler. This guide demonstrates how to manage them directly.

Prerequisites

DPS server running and accessible
dpsctl installed and authenticated
Active topology with entities configured

Basic Resource Group Workflow

Step 1: Create Resource Group

dpsctl resource-group create \
  --resource-group "ml-training-job" \
  --external-id 12345 \
  --policy "Node-High"

Step 2: Add Hardware Resources

dpsctl resource-group add \
  --resource-group "ml-training-job" \
  --entities "node001,node002,node003"

Step 3: (Optional) Configure Per-GPU Power Policies

If your workload requires fine-grained GPU power control, set per-GPU power limits before or after activation:

dpsctl resource-group update \
  --resource-group "ml-training-job" \
  --entity-gpu-policy node001=500,550,600,700,650,700,550,600

Step 4: Activate Power Policies

dpsctl resource-group activate \
  --resource-group "ml-training-job"

Step 5: Cleanup

dpsctl resource-group delete \
  --resource-group "ml-training-job"

Dynamic Resource Management

Resources can be added to or removed from resource groups at any time, including after activation. This enables dynamic workload scaling without full resource group deactivation.

Note: Add/remove operations are rejected while a resource group is activating or deactivating. Wait for the operation to complete before modifying resources.

Adding Resources to an Active Resource Group

When adding resources to an active resource group, policies are applied immediately and power is reallocated as needed:

dpsctl resource-group add \
  --resource-group "ml-training-job" \
  --entities "node004,node005"

Strict Policy Enforcement

Use --strict-policy to ensure the requested policy is applied exactly. If power constraints prevent the policy from being satisfied, the operation fails rather than automatically downgrading:

dpsctl resource-group add \
  --resource-group "ml-training-job" \
  --entities "node004" \
  --policy "Node-High" \
  --strict-policy

Controlling Power Reprovisioning

By default, power may be redistributed from other resource groups if needed (power stealing). Use --allow-reprovision=false to prevent this:

dpsctl resource-group add \
  --resource-group "ml-training-job" \
  --entities "node004" \
  --allow-reprovision=false

Removing Resources from an Active Resource Group

When removing resources from an active resource group, policies on the removed entities are reverted to topology defaults and power is reallocated:

dpsctl resource-group remove \
  --resource-group "ml-training-job" \
  --entities "node004,node005"

Separate Workload Classes

Use separate resource groups when inference and training nodes need different power envelopes. A node must not belong to both groups. Select policies from dpsctl policy list; the names below are examples, not universal defaults.

dpsctl resource-group create \
  --resource-group inference-rg \
  --external-id 1001 \
  --policy Inference-Policy \
  --dpm-enable \
  --prs-enabled \
  --shared-gpu-enable

dpsctl resource-group add \
  --resource-group inference-rg \
  --entities "inference-node-01,inference-node-02"

dpsctl resource-group create \
  --resource-group training-rg \
  --external-id 1002 \
  --policy Training-MaxP \
  --dpm-enable=false \
  --prs-enabled=false

dpsctl resource-group add \
  --resource-group training-rg \
  --entities "training-node-01,training-node-02"

Activate each group with --sync and inspect every node result. The inference group permits PRS to redistribute its shared GPU budget. The training example keeps a fixed policy cap; before activation, ensure all static allocations and unmanaged load fit within every applicable PowerDomain constraint.

If dynamic groups can compete for power, --priority controls stealing and giveback order: a lower numeric value has higher priority. Priority does not reserve capacity or override a PowerDomain or NvGrid limit.

Sandbox a Subset of Nodes

To limit MaxLPS testing to a sandbox while keeping other nodes at MaxP:

Give every production and sandbox ComputerSystem the required MaxP topology-default Policy.
Activate the topology and verify those defaults.
Create a lower-policy MaxLPS resource group containing only sandbox nodes.
Activate it with reprovisioning disabled so the sandbox cannot take power from other resource groups.

For the generic-2-node-topology.json example, both nodes default to power-policy-max. This command temporarily applies power-policy-70 only to the sandbox node:

dpsctl resource-group create \
  --resource-group maxlps-sandbox \
  --external-id 2001 \
  --policy power-policy-70 \
  --dpm-enable \
  --prs-enabled \
  --shared-gpu-enable

dpsctl resource-group add \
  --resource-group maxlps-sandbox \
  --entities "compute-node-02"

dpsctl resource-group activate \
  --resource-group maxlps-sandbox \
  --allow-reprovision=false \
  --sync

Verify that the group contains no production nodes. Removing the node or deleting the resource group restores its power-policy-max topology default. Topology activation still attempts to contact every power-managed node included in that topology; if production BMCs must never be contacted, do not include those nodes in the active topology.

Troubleshooting

Verifying Node Status

After activating a resource group, always verify that each individual node’s policy was applied successfully:

Check that each node in node_statuses has "ok": true
Review any error messages in diag_msg fields
If any nodes show "ok": false, investigate the specific error before proceeding

Example of a failed node:

"node004": {
  "status": {
    "ok": false,
    "diag_msg": "BMC connection timeout"
  }
}

PRS

Power Reservation Steering (PRS) is a product optionally included and directly integrated with DPS that performs real-time power allocation adjustment for resource groups. PRS consumes telemetry provided by DPS to generate recommendations for power allocations at a per-GPU level, which DPS validates against the PDN constraints before applying. PRS can be controlled in a few ways via DPS:

Not deployed (configured via Helm chart values option .Values.prs.enabled)
Toggled via global configuration in the DPS WebUI
Toggled per ResourceGroup during creation with option --prs-enabled=false It is recommended to deploy PRS and leave it enabled globally. For Max-P Resource Group, we recommend disabling PRS and leaving it on for all others.