Resource Groups#

Overview#

Resource groups dynamically associate active-topology resources with a workload. Their policy bundle has higher precedence than the topology bundle, allowing workload-specific behavior without changing the topology selection.

Resource Group Structure#

A resource group contains:

  • Workload identification and metadata.

  • Member compute resources.

  • A persisted policy-bundle selection.

  • Lifecycle and activation state.

  • Optional workload-profile and power-management settings.

  • Relative priority for power stealing and giveback. Lower numbers are higher priority, and the default is 0.

Lifecycle#

Resource groups support these operations:

  1. Create an inactive group.

  2. Add or remove member resources.

  3. Activate the group and apply its effective policies.

  4. Update membership or supported workload settings.

  5. Delete the group and release its resources.

Membership changes on an inactive group are stored without applying hardware policy. Changes on an active group apply or release policy immediately. Membership changes are rejected while activation or deactivation is in progress.

Policy Selection and Precedence#

A topology binding remains the lower-precedence policy source for its managed devices. When a resource group becomes active, DPS composes its higher-precedence binding with the topology binding by policy type. A resource-group policy can supersede or combine with topology policy according to the rules of that type.

An empty resource-group bundle selection resolves to the current global_policy_bundle and the resolved name is persisted on the group. It does not categorically mean “inherit the topology bundle.” If no global default is configured and the resource group contributes no policy of a given type, the topology policy of that type can remain effective.

Create a group with an explicit bundle by using --policy-bundle:

dpsctl resource-group create \
  --resource-group ml-training-job \
  --external-id 12345 \
  --policy-bundle site-balanced

Current resource-group policy-bundle selection is supported at creation time. The server does not currently support changing that selection with dpsctl resource-group update.

When a resource leaves an active group, DPS restores any remaining topology-effective limit. If no effective limit remains, release uses the device idle policy or the hardware-minimum path as described in Policy Lifecycle and Fallback.

Per-GPU Power Limits#

The standalone dpsctl gpu-policy command sets explicit per-GPU watt values for nodes in active resource groups:

dpsctl gpu-policy \
  --node "node001=500,550,600,700,650,700,550,600"

Specify one value for every GPU on the node, in GPU index order. DPS validates the GPU count, device ranges, and available node power budget before applying the update. Refer to dpsctl gpu-policy for command details.

Shared GPU Budget#

Shared GPU mode pools GPU watts across the members of a resource group. DPS enforces the group sum rather than each node’s original compute-policy GPU cap. One node can take more GPU power while another takes less, as long as the group stays inside the resource-group GPU budget.

Omitted shared_gpu_enable values default to on. Pass --shared-gpu-enable=false at create time to opt out. Shared GPU mode can run together with Power Reservation Steering (PRS) when dynamic power management is enabled.

The resource-group GPU budget is a policy-derived ceiling, not the sum of currently allocated GPU caps:

  • When the effective policy sets an explicit GPU limit, that GPU target is the ceiling for the member, and the group ceiling is the sum of those targets.

  • When it does not, each member contributes the greater of its currently allocated GPU watts and the node policy after reserving CPU and memory.

You can raise allocated GPU watts up to that ceiling. The live allocated sum is not a freeze.

When you update GPU policies on a shared-GPU resource group, DPS compares the post-update group total to that ceiling:

  • Nodes in the request contribute their requested per-GPU watts.

  • Nodes that are not in the request contribute their currently allocated per-GPU watts.

  • If the post-update total exceeds the ceiling, DPS clamps feasible increases and can apply lower limits than you requested.

  • If no valid clamp exists, for example when the sum of hardware GPU minima exceeds the remaining budget, DPS rejects the request before it sends device updates.

A request that fits the resource-group GPU budget can still be rejected if the post-clamp aggregate load would violate an ancestor topology constraint. Supported aggregate checks apply to non-compute topology entities such as racks and power distribution network (PDN) devices. Shared-GPU nodes can exceed a per-node GPU policy cap. They still cannot exceed per-GPU hardware min and max limits, and the device or BMC still enforces those limits.

If a topology ancestor is already overloaded, DPS accepts a shared-GPU update when the new load is lower, or when the load is unchanged and the power setpoint is not reduced. An update that increases the overload, or that keeps the same load while lowering the power setpoint, is rejected.

Rejected GPU policy results include a diagnostic that names the violating ancestor, for example shared-GPU update violates topology constraint at device:rack-a. A controller response that would commit an illegal load is also rejected, with a diagnostic such as response violates topology constraints, and is not written to entity state.

Priority#

Resource group --priority orders groups when DPS steals power, and when giveback cannot admit a complete restore of every eligible member. It does not request a policy or reserve watts. Omit the flag to store 0, which is currently the highest precedence. Assign a larger number to groups that can give up power first.

For how to set, inspect, and update priority, and for steal versus giveback order, refer to Resource Group Priority.

Further Reading#