Policies#

group Policies

This chapter describes the methods that handle system policy management and violation settings.

The APIs in Policies module can be broken down into following categories:

Describes APIs for setting up policies and registering callbacks to receive notification in case specific policy condition has been violated.

Functions

dcgmReturn_t dcgmPolicySet(
dcgmHandle_t pDcgmHandle,
dcgmGpuGrp_t groupId,
dcgmPolicy_t *policy,
dcgmStatus_t statusHandle
)#

Set the current violation policy inside the policy manager.

Given the conditions within the dcgmPolicy_t structure, if a violation has occurred, subsequent action(s) may be performed to either report or contain the failure.

Parameters:

  • pDcgmHandle – IN: DCGM Handle

  • groupId – IN: Group ID representing collection of one or more GPUs. Look at dcgmGroupCreate for details on creating the group. Alternatively, pass in the group id as DCGM_GROUP_ALL_GPUS to perform operation on all the GPUs.

  • policy – IN: A reference to dcgmPolicy_t that will be applied to all GPUs in the group.

  • statusHandle – IN/OUT: Resulting status for the operation. Pass it as NULL if the detailed error information is not needed. Refer to dcgmStatusCreate for details on creating a status handle.

Returns:

dcgmReturn_t dcgmPolicyGet(
dcgmHandle_t pDcgmHandle,
dcgmGpuGrp_t groupId,
int count,
dcgmPolicy_t *policy,
dcgmStatus_t statusHandle
)#

Get the current violation policy inside the policy manager.

Given a groupId, a number of policy structures are retrieved.

Parameters:

  • pDcgmHandle – IN: DCGM Handle

  • groupId – IN: Group ID representing collection of one or more GPUs. Look at dcgmGroupCreate for details on creating the group. Alternatively, pass in the group id as DCGM_GROUP_ALL_GPUS to perform operation on all the GPUs.

  • count – IN: The size of the policy array. This is the maximum number of policies that will be retrieved and ultimately should correspond to the number of GPUs specified in the group.

  • policy – OUT: A reference to dcgmPolicy_t that will used as storage for the current policies applied to each GPU in the group.

  • statusHandle – IN/OUT: Resulting status for the operation. Pass it as NULL if the detailed error information for the operation is not needed. Refer to dcgmStatusCreate for details on creating a status handle.

Returns:

dcgmReturn_t dcgmPolicyRegister_v2(
dcgmHandle_t pDcgmHandle,
dcgmGpuGrp_t groupId,
dcgmPolicyCondition_t condition,
fpRecvUpdates callback,
uint64_t userData
)#

Register a function to be called when a specific policy condition (see dcgmPolicyCondition_t) has been violated.

This callback(s) will be called automatically when in DCGM_OPERATION_MODE_AUTO mode and only after dcgmPolicyTrigger when in DCGM_OPERATION_MODE_MANUAL mode. All callbacks are made within a separate thread.

Parameters:

  • pDcgmHandle – IN: DCGM Handle

  • groupId – IN: Group ID representing collection of one or more GPUs. Look at dcgmGroupCreate for details on creating the group. Alternatively, pass in the group id as DCGM_GROUP_ALL_GPUS to perform operation on all the GPUs.

  • condition – IN: The set of conditions specified as an OR’d list (see dcgmPolicyCondition_t) for which to register a callback function

  • callback – IN: A reference to a function that should be called should a violation occur. This function will be called prior to any actions specified by the policy are taken.

  • userData – IN: User data pointer to pass to the userData field of callback

Returns:

dcgmReturn_t dcgmPolicyUnregister(
dcgmHandle_t pDcgmHandle,
dcgmGpuGrp_t groupId,
dcgmPolicyCondition_t condition
)#

Unregister a function to be called for a specific policy condition (see dcgmPolicyCondition_t).

This function will unregister all callbacks for a given condition and handle.

Parameters:

  • pDcgmHandle – IN: DCGM Handle

  • groupId – IN: Group ID representing collection of one or more GPUs. Look at dcgmGroupCreate for details on creating the group. Alternatively, pass in the group id as DCGM_GROUP_ALL_GPUS to perform operation on all the GPUs.

  • condition – IN: The set of conditions specified as an OR’d list (see dcgmPolicyCondition_t) for which to unregister a callback function

Returns:

Describes APIs which can be used to perform direct actions (e.g.

Perform GPU Reset, Run Health Diagnostics) on a group of GPUs.

Functions

dcgmReturn_t dcgmActionValidate(
dcgmHandle_t pDcgmHandle,
dcgmGpuGrp_t groupId,
dcgmPolicyValidation_t validate,
dcgmDiagResponse_v12 *response
)#

Inform the action manager to perform a manual validation of a group of GPUs on the system.

*************************************** DEPRECATED ***************************************

Parameters:

  • pDcgmHandle – IN: DCGM Handle

  • groupId – IN: Group ID representing collection of one or more GPUs. Look at dcgmGroupCreate for details on creating the group. Alternatively, pass in the group id as DCGM_GROUP_ALL_GPUS to perform operation on all the GPUs.

  • validate – IN: The validation to perform after the action.

  • response – OUT: Result of the validation process. Refer to dcgmDiagResponse_t for details.

Returns:

dcgmReturn_t dcgmActionValidate_v2(
dcgmHandle_t pDcgmHandle,
dcgmRunDiag_v10 *drd,
dcgmDiagResponse_v12 *response
)#

Inform the action manager to perform a manual validation of a group of GPUs on the system.

Parameters:

  • pDcgmHandle – IN: DCGM Handle

  • drd – IN: Contains the group id, test names, test parameters, struct version, and the validation that should be performed. Look at dcgmGroupCreate for details on creating the group. Alternatively, pass in the group id as DCGM_GROUP_ALL_GPUS to perform operation on all the GPUs.

  • response – OUT: Result of the validation process. Refer to dcgmDiagResponse_t for details. Note: It’s a caller’s responsibility to make sure the response is zero-initialized, except for the version field.

Returns:

dcgmReturn_t dcgmRunDiagnostic(
dcgmHandle_t pDcgmHandle,
dcgmGpuGrp_t groupId,
dcgmDiagnosticLevel_t diagLevel,
dcgmDiagResponse_v12 *diagResponse
)#

Run a diagnostic on a group of GPUs.

Parameters:

  • pDcgmHandle – IN: DCGM Handle

  • groupId – IN: Group ID representing collection of one or more GPUs. Look at dcgmGroupCreate for details on creating the group. Alternatively, pass in the group id as DCGM_GROUP_ALL_GPUS to perform operation on all the GPUs.

  • diagLevel – IN: Diagnostic level to run

  • diagResponse –

    IN/OUT: Result of running the DCGM diagnostic.

    .version should be set to

    dcgmDiagResponse_version before this call.

Returns:

dcgmReturn_t dcgmStopDiagnostic(dcgmHandle_t pDcgmHandle)#

Stop a diagnostic if there is one currently running.

Parameters:

pDcgmHandle – IN: DCGM Handle

Returns:

Setup and Management#

group Setup and Management

Describes APIs for setting up policies and registering callbacks to receive notification in case specific policy condition has been violated.

Functions

dcgmReturn_t dcgmPolicySet(
dcgmHandle_t pDcgmHandle,
dcgmGpuGrp_t groupId,
dcgmPolicy_t *policy,
dcgmStatus_t statusHandle
)

Set the current violation policy inside the policy manager.

Given the conditions within the dcgmPolicy_t structure, if a violation has occurred, subsequent action(s) may be performed to either report or contain the failure.

Parameters:

  • pDcgmHandle – IN: DCGM Handle

  • groupId – IN: Group ID representing collection of one or more GPUs. Look at dcgmGroupCreate for details on creating the group. Alternatively, pass in the group id as DCGM_GROUP_ALL_GPUS to perform operation on all the GPUs.

  • policy – IN: A reference to dcgmPolicy_t that will be applied to all GPUs in the group.

  • statusHandle – IN/OUT: Resulting status for the operation. Pass it as NULL if the detailed error information is not needed. Refer to dcgmStatusCreate for details on creating a status handle.

Returns:

dcgmReturn_t dcgmPolicyGet(
dcgmHandle_t pDcgmHandle,
dcgmGpuGrp_t groupId,
int count,
dcgmPolicy_t *policy,
dcgmStatus_t statusHandle
)

Get the current violation policy inside the policy manager.

Given a groupId, a number of policy structures are retrieved.

Parameters:

  • pDcgmHandle – IN: DCGM Handle

  • groupId – IN: Group ID representing collection of one or more GPUs. Look at dcgmGroupCreate for details on creating the group. Alternatively, pass in the group id as DCGM_GROUP_ALL_GPUS to perform operation on all the GPUs.

  • count – IN: The size of the policy array. This is the maximum number of policies that will be retrieved and ultimately should correspond to the number of GPUs specified in the group.

  • policy – OUT: A reference to dcgmPolicy_t that will used as storage for the current policies applied to each GPU in the group.

  • statusHandle – IN/OUT: Resulting status for the operation. Pass it as NULL if the detailed error information for the operation is not needed. Refer to dcgmStatusCreate for details on creating a status handle.

Returns:

dcgmReturn_t dcgmPolicyRegister_v2(
dcgmHandle_t pDcgmHandle,
dcgmGpuGrp_t groupId,
dcgmPolicyCondition_t condition,
fpRecvUpdates callback,
uint64_t userData
)

Register a function to be called when a specific policy condition (see dcgmPolicyCondition_t) has been violated.

This callback(s) will be called automatically when in DCGM_OPERATION_MODE_AUTO mode and only after dcgmPolicyTrigger when in DCGM_OPERATION_MODE_MANUAL mode. All callbacks are made within a separate thread.

Parameters:

  • pDcgmHandle – IN: DCGM Handle

  • groupId – IN: Group ID representing collection of one or more GPUs. Look at dcgmGroupCreate for details on creating the group. Alternatively, pass in the group id as DCGM_GROUP_ALL_GPUS to perform operation on all the GPUs.

  • condition – IN: The set of conditions specified as an OR’d list (see dcgmPolicyCondition_t) for which to register a callback function

  • callback – IN: A reference to a function that should be called should a violation occur. This function will be called prior to any actions specified by the policy are taken.

  • userData – IN: User data pointer to pass to the userData field of callback

Returns:

dcgmReturn_t dcgmPolicyUnregister(
dcgmHandle_t pDcgmHandle,
dcgmGpuGrp_t groupId,
dcgmPolicyCondition_t condition
)

Unregister a function to be called for a specific policy condition (see dcgmPolicyCondition_t).

This function will unregister all callbacks for a given condition and handle.

Parameters:

  • pDcgmHandle – IN: DCGM Handle

  • groupId – IN: Group ID representing collection of one or more GPUs. Look at dcgmGroupCreate for details on creating the group. Alternatively, pass in the group id as DCGM_GROUP_ALL_GPUS to perform operation on all the GPUs.

  • condition – IN: The set of conditions specified as an OR’d list (see dcgmPolicyCondition_t) for which to unregister a callback function

Returns:

Policy action and validation behavior#

In DCGM 4.6, dcgmPolicySet() stores the complete dcgmPolicy_t structure, including its action and validation fields. dcgmPolicyGet() returns those stored values. The policy module does not execute them when a condition is violated: it does not reset the GPU, run a diagnostic validation, or emit an action-completion callback.

DCGM_POLICY_ACTION_GPURESET is a deprecated enumeration value. Applications should use policy callbacks as notifications, coordinate any required resource drain or recovery themselves, and invoke diagnostics explicitly.

Setting a policy does not itself start the policy module’s field watches. A registration made with dcgmPolicyRegister_v2() installs the callback watcher and starts policy field collection for the connection. Keep the registration active for as long as policy notifications are required.

Callback registration lifetime#

A callback registered with dcgmPolicyRegister_v2() is not one-shot. The registration remains active after each callback returns and is eligible for later qualifying notifications. It ends when the application calls dcgmPolicyUnregister() or the DCGM connection closes. Callback delivery does not implicitly unregister the callback, so an application does not need to register again after each notification.

Registration persistence does not guarantee a callback for every raw event; notifications remain subject to policy-condition evaluation and notification throttling.

Manual Invocation#

group Manual Invocation

Describes APIs which can be used to perform direct actions (e.g.

Perform GPU Reset, Run Health Diagnostics) on a group of GPUs.

Functions

dcgmReturn_t dcgmActionValidate(
dcgmHandle_t pDcgmHandle,
dcgmGpuGrp_t groupId,
dcgmPolicyValidation_t validate,
dcgmDiagResponse_v12 *response
)

Inform the action manager to perform a manual validation of a group of GPUs on the system.

*************************************** DEPRECATED ***************************************

Parameters:

  • pDcgmHandle – IN: DCGM Handle

  • groupId – IN: Group ID representing collection of one or more GPUs. Look at dcgmGroupCreate for details on creating the group. Alternatively, pass in the group id as DCGM_GROUP_ALL_GPUS to perform operation on all the GPUs.

  • validate – IN: The validation to perform after the action.

  • response – OUT: Result of the validation process. Refer to dcgmDiagResponse_t for details.

Returns:

dcgmReturn_t dcgmActionValidate_v2(
dcgmHandle_t pDcgmHandle,
dcgmRunDiag_v10 *drd,
dcgmDiagResponse_v12 *response
)

Inform the action manager to perform a manual validation of a group of GPUs on the system.

Parameters:

  • pDcgmHandle – IN: DCGM Handle

  • drd – IN: Contains the group id, test names, test parameters, struct version, and the validation that should be performed. Look at dcgmGroupCreate for details on creating the group. Alternatively, pass in the group id as DCGM_GROUP_ALL_GPUS to perform operation on all the GPUs.

  • response – OUT: Result of the validation process. Refer to dcgmDiagResponse_t for details. Note: It’s a caller’s responsibility to make sure the response is zero-initialized, except for the version field.

Returns:

dcgmReturn_t dcgmRunDiagnostic(
dcgmHandle_t pDcgmHandle,
dcgmGpuGrp_t groupId,
dcgmDiagnosticLevel_t diagLevel,
dcgmDiagResponse_v12 *diagResponse
)

Run a diagnostic on a group of GPUs.

Parameters:

  • pDcgmHandle – IN: DCGM Handle

  • groupId – IN: Group ID representing collection of one or more GPUs. Look at dcgmGroupCreate for details on creating the group. Alternatively, pass in the group id as DCGM_GROUP_ALL_GPUS to perform operation on all the GPUs.

  • diagLevel – IN: Diagnostic level to run

  • diagResponse –

    IN/OUT: Result of running the DCGM diagnostic.

    .version should be set to

    dcgmDiagResponse_version before this call.

Returns:

dcgmReturn_t dcgmStopDiagnostic(dcgmHandle_t pDcgmHandle)

Stop a diagnostic if there is one currently running.

Parameters:

pDcgmHandle – IN: DCGM Handle

Returns: