Policies#

group Policies

This chapter describes the methods that handle system policy management and violation settings.

The APIs in Policies module can be broken down into following categories:

Setup and Management#

group Setup and Management

Describes APIs for setting up policies and registering callbacks to receive notification in case specific policy condition has been violated.

Functions

dcgmReturn_t dcgmPolicySet(
dcgmHandle_t pDcgmHandle,
dcgmGpuGrp_t groupId,
dcgmPolicy_t *policy,
dcgmStatus_t statusHandle
)#

Set the current violation policy inside the policy manager.

Given the conditions within the dcgmPolicy_t structure, if a violation has occurred, subsequent action(s) may be performed to either report or contain the failure.

Deprecated:

For threshold monitoring, use the field-policy APIs beginning with dcgmPolicyCreate.

Parameters:

  • pDcgmHandle – IN: DCGM Handle

  • groupId – IN: Group ID representing collection of one or more GPUs. Look at dcgmGroupCreate for details on creating the group. Alternatively, pass in the group id as DCGM_GROUP_ALL_GPUS to perform operation on all the GPUs.

  • policy – IN: A reference to dcgmPolicy_t that will be applied to all GPUs in the group.

  • statusHandle – IN/OUT: Resulting status for the operation. Pass it as NULL if the detailed error information is not needed. Refer to dcgmStatusCreate for details on creating a status handle.

Returns:

dcgmReturn_t dcgmPolicyGet(
dcgmHandle_t pDcgmHandle,
dcgmGpuGrp_t groupId,
int count,
dcgmPolicy_t *policy,
dcgmStatus_t statusHandle
)#

Get the current violation policy inside the policy manager.

Given a groupId, a number of policy structures are retrieved.

Deprecated:

For field-policy state, use dcgmPolicyGetOne or dcgmPolicyGetAll.

Parameters:

  • pDcgmHandle – IN: DCGM Handle

  • groupId – IN: Group ID representing collection of one or more GPUs. Look at dcgmGroupCreate for details on creating the group. Alternatively, pass in the group id as DCGM_GROUP_ALL_GPUS to perform operation on all the GPUs.

  • count – IN: The size of the policy array. This is the maximum number of policies that will be retrieved and ultimately should correspond to the number of GPUs specified in the group.

  • policy – OUT: A reference to dcgmPolicy_t that will used as storage for the current policies applied to each GPU in the group.

  • statusHandle – IN/OUT: Resulting status for the operation. Pass it as NULL if the detailed error information for the operation is not needed. Refer to dcgmStatusCreate for details on creating a status handle.

Returns:

dcgmReturn_t dcgmPolicyCreate(
dcgmHandle_t pDcgmHandle,
unsigned short fieldId,
const char *name,
double threshold,
dcgmPolicyOperator_t policyOperator,
const dcgmGroupEntityPair_t *entities,
unsigned int entityCount,
unsigned int channelMask,
unsigned int enabled,
unsigned int rateLimitSec,
uint64_t *policyId
)#

Create a field-based policy that monitors a DCGM field against a threshold.

This call blocks until the policy module responds. The module assigns policyId and persists the policy when watch setup succeeds.

Parameters:

  • pDcgmHandle – IN: DCGM Handle

  • fieldId – IN: DCGM field ID to monitor

  • name – IN: NUL-terminated policy display name

  • threshold – IN: Comparison threshold

  • policyOperator – IN: Comparison operator (see dcgmPolicyOperator_t)

  • entities – IN: GPU entities to monitor; may be NULL when entityCount is 0

  • entityCount – IN: Number of valid entries in entities

  • channelMask – IN: Bitmask of notification channels (see dcgmPolicyChannel_t)

  • enabled – IN: Non-zero to create the policy enabled; zero to create disabled

  • rateLimitSec – IN: Minimum seconds between notifications

  • policyId – OUT: Module-assigned policy identifier on success

Returns:

dcgmReturn_t dcgmPolicyModify(
dcgmHandle_t pDcgmHandle,
const dcgmPolicyInfo_t *policy
)#

Modify an existing field-based policy identified by policyId and fieldId.

This call blocks until the policy module responds. Runtime fields such as violationCount are preserved by the module.

Parameters:

  • pDcgmHandle – IN: DCGM Handle

  • policy – IN: Policy configuration to apply; version must be dcgmPolicyInfo_version

Returns:

dcgmReturn_t dcgmPolicyDelete(
dcgmHandle_t pDcgmHandle,
unsigned short fieldId,
uint64_t policyId,
unsigned int deleteAll
)#

Delete one field-based policy or all field-based policies.

This call blocks until the policy module responds.

Parameters:

  • pDcgmHandle – IN: DCGM Handle

  • fieldId – IN: Field ID of the policy to delete when deleteAll is 0

  • policyId – IN: Policy identifier to delete when deleteAll is 0

  • deleteAll – IN: Non-zero to delete all field-based policies; zero to delete one policy

Returns:

dcgmReturn_t dcgmPolicyEnable(
dcgmHandle_t pDcgmHandle,
unsigned short fieldId,
uint64_t policyId
)#

Enable a field-based policy.

This call blocks until the policy module responds and field watches are refreshed.

Parameters:

  • pDcgmHandle – IN: DCGM Handle

  • fieldId – IN: Field ID of the policy to enable

  • policyId – IN: Policy identifier to enable

Returns:

dcgmReturn_t dcgmPolicyDisable(
dcgmHandle_t pDcgmHandle,
unsigned short fieldId,
uint64_t policyId
)#

Disable a field-based policy.

This call blocks until the policy module responds and field watches are refreshed.

Parameters:

  • pDcgmHandle – IN: DCGM Handle

  • fieldId – IN: Field ID of the policy to disable

  • policyId – IN: Policy identifier to disable

Returns:

dcgmReturn_t dcgmPolicyGetOne(
dcgmHandle_t pDcgmHandle,
unsigned short fieldId,
uint64_t policyId,
dcgmPolicyInfo_t *policy
)#

Get one field-based policy by fieldId and policyId.

This call blocks until the policy module responds.

Parameters:

  • pDcgmHandle – IN: DCGM Handle

  • fieldId – IN: Field ID of the policy to retrieve

  • policyId – IN: Policy identifier to retrieve

  • policy – IN/OUT: Caller buffer; version must be dcgmPolicyInfo_version. Populated on success.

Returns:

dcgmReturn_t dcgmPolicyGetAll(
dcgmHandle_t pDcgmHandle,
dcgmPolicyInfo_t *policies,
unsigned int *count
)#

Get all registered field-based policies.

This call blocks until the policy module responds. On DCGM_ST_INSUFFICIENT_SIZE, policies contains up to count entries and count is updated to the total available.

Parameters:

  • pDcgmHandle – IN: DCGM Handle

  • policies – OUT: Array of policy records; may be NULL when count is 0

  • count – IN/OUT: On input, capacity of policies. On output, total policies available.

Returns:

dcgmReturn_t dcgmPolicyImport(
dcgmHandle_t pDcgmHandle,
const char *path
)#

Import field-based policies from a YAML file.

This call blocks until the policy module responds and field watches are refreshed.

Parameters:

  • pDcgmHandle – IN: DCGM Handle

  • path – IN: NUL-terminated path to the YAML policy file

Returns:

dcgmReturn_t dcgmPolicyGetViolations(
dcgmHandle_t pDcgmHandle,
int64_t sinceTimestamp,
dcgm_policy_violation_t *violations,
unsigned int *count
)#

Read recent field-based policy violations.

Note

Thread-safe for distinct handles. Does not throw C++ exceptions.

Parameters:

  • pDcgmHandle – IN: DCGM handle

  • sinceTimestamp – IN: Return violations at or after this timestamp (usec since 1970); 0 for all recent

  • violations – OUT: Caller-allocated array of dcgm_policy_violation_t

  • count – IN/OUT: On input, capacity of violations; on output, total violations available

Returns:

dcgmReturn_t dcgmPolicyRegister_v3(
dcgmHandle_t pDcgmHandle,
unsigned short fieldId,
uint64_t policyId,
fpRecvPolicyViolation callback,
uint64_t userData
)#

Register a callback for field-based policy violations.

Note

Callbacks are invoked from a dedicated policy thread. Thread-safe for distinct handles. Does not throw C++ exceptions.

Parameters:

  • pDcgmHandle – IN: DCGM handle

  • fieldId – IN: Field ID to monitor; pass 0 to receive all field-policy violations

  • policyId – IN: Policy identifier; pass 0 to receive violations for all policies on fieldId

  • callback – IN: Callback invoked when a matching violation occurs

  • userData – IN: User data passed to callback

Returns:

dcgmReturn_t dcgmPolicyUnregister_v3(
dcgmHandle_t pDcgmHandle,
unsigned short fieldId,
uint64_t policyId
)#

Unregister a field-based policy violation callback.

Note

Thread-safe for distinct handles. Does not throw C++ exceptions.

Parameters:
Returns:

dcgmReturn_t dcgmPolicyRegister_v2(
dcgmHandle_t pDcgmHandle,
dcgmGpuGrp_t groupId,
dcgmPolicyCondition_t condition,
fpRecvUpdates callback,
uint64_t userData
)#

Register a function to be called when a specific policy condition (see dcgmPolicyCondition_t) has been violated.

This callback(s) will be called automatically when in DCGM_OPERATION_MODE_AUTO mode and only after dcgmPolicyTrigger when in DCGM_OPERATION_MODE_MANUAL mode. All callbacks are made within a separate thread.

Deprecated:

This group-policy callback API is being retired. Use the field-policy API: create a policy with dcgmPolicyCreate and register a callback with dcgmPolicyRegister_v3.

Parameters:

  • pDcgmHandle – IN: DCGM Handle

  • groupId – IN: Group ID representing collection of one or more GPUs. Look at dcgmGroupCreate for details on creating the group. Alternatively, pass in the group id as DCGM_GROUP_ALL_GPUS to perform operation on all the GPUs.

  • condition – IN: The set of conditions specified as an OR’d list (see dcgmPolicyCondition_t) for which to register a callback function

  • callback – IN: A reference to a function that should be called should a violation occur. This function will be called prior to any actions specified by the policy are taken.

  • userData – IN: User data pointer to pass to the userData field of callback

Returns:

dcgmReturn_t dcgmPolicyUnregister(
dcgmHandle_t pDcgmHandle,
dcgmGpuGrp_t groupId,
dcgmPolicyCondition_t condition
)#

Unregister a function to be called for a specific policy condition (see dcgmPolicyCondition_t).

This function will unregister all callbacks for a given condition and handle.

Deprecated:

This group-policy callback API is being retired. Use dcgmPolicyUnregister_v3 to unregister a field-policy callback registered with dcgmPolicyRegister_v3.

Parameters:

  • pDcgmHandle – IN: DCGM Handle

  • groupId – IN: Group ID representing collection of one or more GPUs. Look at dcgmGroupCreate for details on creating the group. Alternatively, pass in the group id as DCGM_GROUP_ALL_GPUS to perform operation on all the GPUs.

  • condition – IN: The set of conditions specified as an OR’d list (see dcgmPolicyCondition_t) for which to unregister a callback function

Returns:

Field-based policy workflow#

Applications should use the field-based policy interface. Pass the field, threshold, operator, entity list, notification channels, initial state, and rate limit directly to dcgmPolicyCreate(); DCGM returns the module-owned policy ID. dcgmPolicyModify() accepts a dcgmPolicyInfo_t containing the field ID and policy ID of the policy to change. Those two IDs also identify a policy for dcgmPolicyDelete(), dcgmPolicyEnable(), dcgmPolicyDisable(), and dcgmPolicyGetOne(). Use dcgmPolicyGetAll() to enumerate policies and dcgmPolicyImport() to load the documented YAML format.

Violations can be retrieved with dcgmPolicyGetViolations() or delivered through a callback registered with dcgmPolicyRegister_v3(). A v3 registration is connection-bound and remains active until dcgmPolicyUnregister_v3() is called or the DCGM connection closes.

The field-policy operators, notification channels, configuration structures, and violation structure are defined in dcgm_structs.h. For command-line examples and the YAML schema, see dcgmi policy.

Policy action and validation behavior#

The following behavior applies to the deprecated legacy group-policy API. dcgmPolicySet() stores the complete dcgmPolicy_t structure, including its action and validation fields. dcgmPolicyGet() returns those stored values. The policy module does not execute them when a condition is violated: it does not reset the GPU, run a diagnostic validation, or emit an action-completion callback.

DCGM_POLICY_ACTION_GPURESET is a deprecated enumeration value. Applications should use policy callbacks as notifications, coordinate any required resource drain or recovery themselves, and invoke diagnostics explicitly.

Setting a policy does not itself start the policy module’s field watches. A registration made with dcgmPolicyRegister_v2() installs the callback watcher and starts policy field collection for the connection. Keep the registration active for as long as policy notifications are required.

Callback registration lifetime#

A callback registered with dcgmPolicyRegister_v2() is not one-shot. The registration remains active after each callback returns and is eligible for later qualifying notifications. It ends when the application calls dcgmPolicyUnregister() or the DCGM connection closes. Callback delivery does not implicitly unregister the callback, so an application does not need to register again after each notification.

Registration persistence does not guarantee a callback for every raw event; notifications remain subject to policy-condition evaluation and notification throttling.

Manual Invocation#

group Manual Invocation

Describes APIs which can be used to perform direct actions (e.g.

Perform GPU Reset, Run Health Diagnostics) on a group of GPUs.

Functions

dcgmReturn_t dcgmActionValidate(
dcgmHandle_t pDcgmHandle,
dcgmGpuGrp_t groupId,
dcgmPolicyValidation_t validate,
dcgmDiagResponse_v12 *response
)#

Inform the action manager to perform a manual validation of a group of GPUs on the system.

*************************************** DEPRECATED ***************************************

Parameters:

  • pDcgmHandle – IN: DCGM Handle

  • groupId – IN: Group ID representing collection of one or more GPUs. Look at dcgmGroupCreate for details on creating the group. Alternatively, pass in the group id as DCGM_GROUP_ALL_GPUS to perform operation on all the GPUs.

  • validate – IN: The validation to perform after the action.

  • response – OUT: Result of the validation process. Refer to dcgmDiagResponse_t for details.

Returns:

dcgmReturn_t dcgmActionValidate_v2(
dcgmHandle_t pDcgmHandle,
dcgmRunDiag_v10 *drd,
dcgmDiagResponse_v12 *response
)#

Inform the action manager to perform a manual validation of a group of GPUs on the system.

Parameters:

  • pDcgmHandle – IN: DCGM Handle

  • drd – IN: Contains the group id, test names, test parameters, struct version, and the validation that should be performed. Look at dcgmGroupCreate for details on creating the group. Alternatively, pass in the group id as DCGM_GROUP_ALL_GPUS to perform operation on all the GPUs.

  • response – OUT: Result of the validation process. Refer to dcgmDiagResponse_t for details. Note: It’s a caller’s responsibility to make sure the response is zero-initialized, except for the version field.

Returns:

dcgmReturn_t dcgmRunDiagnostic(
dcgmHandle_t pDcgmHandle,
dcgmGpuGrp_t groupId,
dcgmDiagnosticLevel_t diagLevel,
dcgmDiagResponse_v12 *diagResponse
)#

Run a diagnostic on a group of GPUs.

Parameters:

  • pDcgmHandle – IN: DCGM Handle

  • groupId – IN: Group ID representing collection of one or more GPUs. Look at dcgmGroupCreate for details on creating the group. Alternatively, pass in the group id as DCGM_GROUP_ALL_GPUS to perform operation on all the GPUs.

  • diagLevel – IN: Diagnostic level to run

  • diagResponse –

    IN/OUT: Result of running the DCGM diagnostic.

    .version should be set to

    dcgmDiagResponse_version before this call.

Returns:

dcgmReturn_t dcgmStopDiagnostic(dcgmHandle_t pDcgmHandle)#

Stop a diagnostic if there is one currently running.

Parameters:

pDcgmHandle – IN: DCGM Handle

Returns: