Resource Group Priority#

Resource group priority tells DPS which workloads to reduce first when power must be taken back, and which restore tier to prefer if a complete giveback cannot be admitted. Use it to express relative importance across resource groups. Do not use it as a reserved watt budget, a policy, or a guarantee that a group keeps its requested allocation.

Operator Contract#

The following table is the public contract for --priority.

Item	Operator meaning
Numeric direction	A lower number is higher priority.
Default	If you omit `--priority`, the value is `0`. That is currently the highest possible precedence.
Valid range	An integer from `0` through `2147483647`.
Giveback	DPS first tries one complete restore of all eligible throttled members. Priority tiers are used only when that complete plan cannot be admitted.
Stealing and reduction	Eligible groups are reduced in reverse-priority bands: higher numbers first. Groups that share a number are reduced together.
Duplicate values	Duplicate priorities are valid. Product order among equal-priority groups is unspecified.
What priority does not do	Priority does not request watts, set a policy, define a floor, or start a background rebalance loop.

Warning

Assigning --priority 1 (or any positive value) ranks that group below every group that still uses the default of 0. If production workloads omit the flag and a batch group sets 10, the batch group is reduced first. If you intend the opposite, set an explicit higher number on the less important groups, or set 0 only on the groups that must be served first.

When to Set Priority#

Set priority when two or more resource groups can compete for the same constrained power, and you care which one is reduced last. Typical cases include:

A production inference tier that must keep power longer than a research or batch tier during an NvGrid load target.
Several static resource groups that share a feed, where importance is independent of the policy each group requested.
Scheduler-created groups that map quality-of-service classes onto relative power importance.

Prefer a different control when the actual need is not ordering:

Need	Control to use
A higher or lower requested allocation	Resource group or entity power policy, not a smaller priority number.
Refuse a reduced allocation on this group	`--strict-policy` on activate or add.
Prevent taking power from other groups during this activate or add	`--allow-reprovision=false` on activate or add.
Stay off the grid or excursion victim list	Do not rely on the `dpm_enable` list field. Steal eligibility is an effective DPM policy with a specified strategy.

Priority is independent of policy-bundle selection. The bundle supplies the effective power controls; priority orders steal victims, and giveback tiers only after a complete restore cannot be admitted.

Three-Group Example#

The following example uses three names so default 0, an explicit production value, and a lower-importance value are not conflated.

Node-High, Node-Med, and Node-Low are operator-created bundles, used here only to keep the three groups distinguishable. DPS ships MaxP, MaxQ, and MaxLPS, which select power modes rather than priority tiers. Create the bundles with dpsctl policy bundle upsert before you run this example, or substitute names that already exist on your server.

# production-serving uses the default. Omitting --priority stores 0,
# which is the highest precedence.
dpsctl resource-group create \
  --resource-group production-serving \
  --external-id 1 \
  --policy-bundle MaxP

# interactive-dev is important, but ranks below any group left at 0.
dpsctl resource-group create \
  --resource-group interactive-dev \
  --external-id 2 \
  --policy-bundle MaxP \
  --priority 10

# batch-research is reduced first when power is scarce.
dpsctl resource-group create \
  --resource-group batch-research \
  --external-id 3 \
  --policy-bundle MaxP \
  --priority 20

Add entities, then activate. Setting priority at create time only persists the value. Redistribution runs later, when a trigger such as activation with reprovisioning, an NvGrid load target, or excursion mitigation needs to move power.

dpsctl resource-group add --resource-group production-serving --entities "node001,node002"
dpsctl resource-group add --resource-group interactive-dev --entities "node003,node004"
dpsctl resource-group add --resource-group batch-research --entities "node005,node006"

dpsctl resource-group activate --resource-group production-serving --sync
dpsctl resource-group activate --resource-group interactive-dev --sync
dpsctl resource-group activate --resource-group batch-research --sync

Inspect the stored values with dpsctl resource-group list. Confirm that priority is 0, 10, and 20 respectively, that policy_bundle is MaxP, and that activation status matches what you expect.

The following table shows steal order. Giveback is not the reverse of this table on the first attempt. DPS first tries to restore every eligible throttled member together. Priority decides restore order only if that complete plan cannot be admitted, and that fallback order is the reverse of steal order: lowest number first.

Event	First	Then	Last
Stealing or grid reduction	`batch-research` (`20`) reduced	`interactive-dev` (`10`)	`production-serving` (`0`)
Fallback giveback	`production-serving` (`0`) restored	`interactive-dev` (`10`)	`batch-research` (`20`)

When a tighter load target or activation steal needs watts, reductions start at the least important eligible band. When spare power returns, eligible throttled groups recover together if the complete plan fits. If it does not, restoration walks the bands in reverse of steal order.

Create, Inspect, and Update#

Create#

dpsctl resource-group create --priority defaults to 0. The flag is optional.

dpsctl resource-group create \
  --resource-group example-low \
  --external-id 10 \
  --priority 100

Preconditions: you are authenticated, and the resource group name is unique.

Verify: dpsctl resource-group list reports "priority": 100 for example-low. Create does not apply device policies. Add entities and activate before you treat allocation as applied.

List#

dpsctl resource-group list

Use list output to confirm the stored priority field, activation status, and policy_bundle. The retained policy_name field is compatibility output; it is not the selected policy bundle. A successful create or update RPC is not evidence that devices received new limits. The dpm_enable field is not the steal-eligibility check.

If the priority field is missing from the output of an older dpsctl binary, the installed client may predate the flag. Confirm versions before you troubleshoot ordering:

dpsctl --version
dpsctl server-version

Align the client and server so both report the same version. Refer to the dpsctl command reference for --no-version-check and related global flags.

Update#

dpsctl resource-group update --priority sends a priority change only when you pass --priority. Omitting the flag leaves the stored value unchanged.

dpsctl resource-group update --resource-group interactive-dev --priority 5
dpsctl resource-group list

Updating priority does not by itself steal or give back power. The new value is used the next time a giveback or stealing path runs.

Activate and Add Triggers#

Activation and adding entities to an active group can steal from other groups when there is not enough power and --allow-reprovision is true (the default).

dpsctl resource-group activate \
  --resource-group production-serving \
  --allow-reprovision \
  --sync

Use --allow-reprovision=false when the new group must not lower other groups. Use --strict-policy when this group’s requested allocation must apply exactly; if power is insufficient, the request fails instead of accepting a reduced allocation.

For flag details, refer to resource-group create, update, list, activate, and add.

Giveback Versus Stealing#

Stealing uses reverse-priority bands. Giveback uses the reverse of that band order only after a complete restore plan fails.

sequenceDiagram accTitle: Priority during steal and giveback accDescr: Operators persist a priority value. Stealing reduces higher numbers first. Giveback tries a complete restore before walking priority tiers. actor Operator participant dpsctl participant DPS Operator->>dpsctl: create or update RG with --priority N dpsctl->>DPS: Persist priority Note over DPS: Setting N does not rebalance by itself. alt Activation, add, NvGrid, or excursion needs power DPS->>DPS: Steal higher N bands first, share within a band DPS->>Operator: Lower-priority eligible groups reduced first else Spare power is available DPS->>DPS: Try one complete restore of all eligible members alt Complete plan admits DPS->>Operator: Eligible throttled groups recover together else Complete plan is rejected DPS->>DPS: Restore lower N tiers first DPS->>Operator: Higher-priority tiers recover first end end

Giveback#

Giveback runs when power is freed, for example when:

An NvGrid load target expires or is replaced with a higher limit.
Excursion mitigation succeeds and headroom returns.
Resources are removed from a group, or a group is deactivated or deleted.

DPS first tries one complete restore of every eligible throttled member. If that plan fits topology limits, it is dispatched and priority does not pick a winner. If the complete plan cannot be admitted, DPS walks priority tiers from highest priority (lowest number) to lowest and restores one tier at a time. Equal-priority members in a fallback tier are restored together. Priority does not create watts. Giveback can stop with residual unused power when the remaining headroom is smaller than the next valid step.

Stealing and Grid Redistribution#

Stealing runs when a request or mitigation needs power that is not free, for example when:

You activate a group, or add entities to an active group, with --allow-reprovision (default true).
NvGrid applies a tighter load target.
Excursion mitigation reduces groups that have an effective DPM policy under the excursion.

Eligible victims are grouped by --priority. Higher numbers (lower importance) are reduced first. Groups that share a number are reduced together rather than draining one peer completely before touching another. DPS lowers policy or caps within approved floors. Grid and excursion reduction are best effort: partial progress can remain if the target cannot be met without terminating jobs. Resource-group activation stealing is request-scoped: if the activation cannot complete, stolen changes to other groups are rolled back.

Steal eligibility is not the dpm_enable field on dpsctl resource-group list. A group is a candidate when its effective policy assignment includes a DPM policy with a specified strategy. Non-workload groups also need an aggregate power policy.

--dpm-enable=false opts the group out of dynamic power management, and as a side effect DPS then selects that group’s own policy strictly, the way --strict-policy does for a single request. Neither flag keeps the group off a later grid or excursion victim list.

NvGrid load targets still do not terminate jobs. If every eligible group is already at its floor, the target can remain unmet (compliant=false). Refer to NvGrid and Excursion Mitigation.

Duplicate Priorities#

You can assign the same number to more than one group. Product behavior requires ordered levels for different numbers. It does not promise which equal-priority group is chosen first.

When several groups share a priority during stealing, DPS reduces that band together rather than draining one peer completely before touching another. Giveback fallback tiers restore equal-priority members together after a complete restore plan is rejected. Do not automate against device count, requested watts, external ID, or resource group ID as tie-breaks. Those comparators are not the live steal or giveback path. If you need a deterministic winner across groups, give them different priority numbers.

Floors, Strict Mode, and Best Effort#

Priority never defines a floor. The following bounds are independent:

Bound	Meaning
Requested allocation	The effective limits selected from the resource group’s policy bundle, plus any standalone per-GPU caps.
Effective-policy floor	The lowest limit allowed by the effective policies while lowering a group during ordinary stealing.
Device or hardware minimum	The hardware lower bound. Grid and excursion paths can escalate to this bound after policy floors are exhausted.
Shared-GPU minimum	Additional lower bound when shared-GPU behavior is enabled on the group.
Strict request	`--strict-policy` applies to the requesting group. It fails the request instead of accepting a reduced allocation for that group. It does not stop stealing from other groups when `--allow-reprovision` is true.

When no reducible headroom remains above the applicable floors, that group is skipped as a further victim. Grid mitigation keeps whatever reduction already applied (best effort) and reports the outcome. Activation stealing that cannot satisfy the request rolls victim changes back. Floor-violation exception alerts are not a current operator-facing guarantee.

Observe Outcomes#

After a redistribution event, do not stop at the create or update status. Confirm:

dpsctl resource-group list shows the expected priority, activation_status, and policy_bundle.
Activate or add responses (or dpsctl resource-group status for async work) show per-node policy apply results.
For NvGrid events, dpsctl nvgrid get-current and webhook payloads show the active load target and compliance.
Device power caps moved in the direction you expect: lower-priority bands dropped first under a tighter target. After the target relaxes, eligible groups may recover together. Higher-priority tiers recover first only when a complete restore cannot be admitted.

Troubleshooting#

Important workloads were reduced first. Check list output. Groups that omitted --priority are 0 and outrank every positive value. Raise the number on less important groups, or set 0 only on the groups that must win.

Every group is 0, and order looks arbitrary. That is expected. Duplicate default priority is valid and unordered as a product contract. Assign distinct numbers if you need a defined sequence.

You changed --priority and nothing moved. Priority is consulted when giveback or stealing runs. Activate, add with reprovisioning, apply or expire an NvGrid target, or wait for an excursion path. Updating the field alone does not rebalance.

A high-priority group still dropped. Priority orders eligible victims. It does not reserve watts. If the higher-priority group is the only remaining candidate, or every lower-priority group is already at a floor, DPS can still reduce it. Confirm applied limits versus the bundle-derived request, whether the group has an effective DPM strategy, and whether --strict-policy was set on the requesting operation.

A lower-priority group recovered at the same time as a higher-priority group. That can be correct. Giveback prefers one complete restore of all eligible members. Priority tiers apply only when that complete plan cannot be admitted.

A group with dpm_enable false was still reduced. Steal eligibility is the effective DPM policy strategy, not the dpm_enable list field. --dpm-enable=false opts the group out of dynamic power management and makes its own policy selection strict. It does not remove the group from later grid or excursion victim selection.

Activation failed after stealing from others. Request-scoped stealing rolls back other groups when the activation cannot complete. Inspect the activate response and dpsctl resource-group status. Retry with a bundle that requests lower limits, more --allow-reprovision headroom, or --strict-policy omitted if a reduced allocation is acceptable for the new group.

--priority is not in dpsctl resource-group create --help. The client is older than the server feature. Upgrade dpsctl and confirm dpsctl --version matches dpsctl server-version.