Policy#

The policy module is module ID 5. It watches a defined set of GPU conditions and sends notifications to registered clients when those conditions occur. The module is loaded when policy functionality is first requested.

When policy is useful#

Policy is useful when an operator already knows the condition that matters and wants event-driven handling rather than periodic polling. Examples include notifying a node agent when a double-bit ECC error occurs, watching for XID or NVLink errors during a reproducer, or feeding a scheduler-aware recovery service after a serious error.

Policy is deliberately narrower than a general rules engine. The available conditions and thresholds are built into DCGM. For response logic—such as draining a scheduler node, resetting a GPU, running diagnostics, paging an on-call engineer, or correlating several metrics—an external controller should consume DCGM health, fields, or policy notifications and perform that automation.

How policy fits with other features#

Policy, health monitoring, and diagnostics are related but not interchangeable:

Health Monitoring evaluates a broad set of retained telemetry and returns a current health summary when queried.
Policy waits for one of seven configured event or threshold conditions and notifies a connected client. In DCGM 4.6 it does not reset a GPU or run a diagnostic from its stored action and validation fields.
Diagnostics actively exercises an available GPU when invoked explicitly by an operator or management integration.

A policy is associated with the GPUs in a DCGM group. The setting remains in the running host engine after the configuring dcgmi process exits, but it is not durable across a host-engine restart. Setting a policy replaces the selected group’s previous complete policy.

Conditions and stored response fields#

The command-line interface exposes these conditions:

Policy conditions#
Condition	Threshold	Typical reason to use it
Retired pages	Caller-supplied page count	Escalate when accumulated memory faults cross a site limit.
GPU temperature	Caller-supplied degrees Celsius	Observe a workload or cooling problem at a chosen temperature.
GPU power	Caller-supplied watts	Observe operation at or above a chosen power level.
Double-bit ECC errors	Internal criterion	React to an uncorrectable memory error.
PCIe errors or replays	Internal criterion	React to bus errors or excessive replay behavior.
NVLink errors	Internal criterion	React to an interconnect error.
XID errors	Driver event	React to a driver-reported GPU error.

Any selected condition can produce a notification. The --set syntax also requires action and validation values inherited from the policy API:

Action 0 represents no action. Action 1 is stored as the deprecated DCGM_POLICY_ACTION_GPURESET value.
Validation 0 represents no validation. Values 1, 2, and 3 are stored as short, medium, and long validation selections.

DCGM 4.6 stores and returns these fields but the policy module does not execute them. A nonzero action does not reset a GPU, and a nonzero validation does not run diagnostics. Use 0,0 for notification workflows so the displayed policy does not imply an automatic response that will not occur.

Example: notification-only temperature and ECC policy#

Suppose group 2 represents GPUs assigned to a debugging session. Configure a policy that takes no automatic action and runs no validation, but triggers on either a temperature of 85 degrees Celsius or a double-bit ECC error:

$ dcgmi policy --set 0,0 --group 2 --maxtemp 85 --eccerrors
$ dcgmi policy --get --group 2 --verbose

The first zero selects no action; the second selects no post-action validation. --get --verbose verifies each GPU rather than relying on a homogeneous group summary.

In a terminal that will remain attached, register for notifications:

$ dcgmi policy --reg --group 2

The listener blocks until interrupted and prints violations as they arrive. The current dcgmi implementation registers the listener for every supported policy condition; the conditions supplied to --set determine the stored policy, while --reg controls this connected observer and starts the policy module’s field watches. --set alone stores the policy but does not start those watches. A long-running management integration normally uses the programmatic callback interface rather than keeping an interactive terminal open.

Clear the policy when the session ends:

$ dcgmi policy --clear --group 2

Automate recovery in the notification consumer#

Use action 0 and validation 0, then consume policy notifications in a scheduler-aware service. The service can drain the resource, perform the site-approved recovery operation, and invoke diagnostics after the GPU is available.

Although --set 1,1 remains accepted and --get displays the stored GPU reset and short-validation selections, DCGM 4.6 does not perform either operation and does not send a second action-completion notification.

Interpreting policy state#

The policy applies independently to GPUs in the group. If a summary reports non-homogeneous settings, use policy --get --verbose to see which GPU differs. Adding a condition with another --set does not merge it with the old policy; include every desired condition in the replacement command. Action and validation values in this output describe stored legacy fields, not operations that the policy module performed.

Thresholds are trigger values, not configuration limits. For example, --maxtemp 85 does not change a GPU thermal limit, and --maxpower 300 does not set a 300 W power cap. Use Configuration to change supported GPU configuration properties.